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Abstract 
Although there is a growing body of evidence to support the use of direct behavior rating (DBR) 
as a formative behavioral assessment method in school-based problem-solving models, this work 
has centered largely on the assessment of attending and problem behaviors and on the use of 
single-item DBR scales. The primary purpose of the current study was to report on the 
development and initial validation of teacher-completed multi-item DBR scales (DBR-MIS) 
designed to assess three constructs that represent behaviors that are widely considered to support 
student academic achievement (Academic Engagement, Interpersonal Skills, and Study Skills). 
Development of the scales involved a content validity study utilizing feedback from a panel of 
consumers (teachers, parents, and school administrators) and a panel of researchers with 
expertise in school-based behavioral assessment. Results of an exploratory factor analysis of 
ratings completed by teachers in Grades K through 3 (N = 307) supported a one-factor solution 
for each of the aforementioned constructs. Favorable internal consistency was found for each 


scale. 
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Content Validation of Direct Behavior Rating Multi-Item Academic Enabler Scales 
Using Exploratory Factor Analysis 

Over the past several decades school-based assessment has shifted from a primarily 
reactive and summative orientation to a proactive, problem-solving framework in which early 
identification (screening) and repeated formative measurement (progress monitoring) are 
emphasized (Tilly, 2008). This transition has been spurred in part by changes in federal policy, 
such as the reauthorization of the Individuals with Disabilities Act (IDEA; 2004) and more 
recently the Every Student Succeeds Act (ESSA; 2015). In recent years the percentage of school 
districts reporting implementation of multi-tiered systems of support (MTSS) for students with 
social-emotional and behavioral (SEB) problems has risen dramatically (Spectrum K12 School 
Solutions, 2011; Zirkel, 2011). In addition to effective screening assessments for early 
identification, valid and feasible progress monitoring assessments are a vital component of 
effective MTSS (National Center on Response to Intervention, 2010; Sugai & Horner, 2009). On 
both systems- and individual- levels, progress monitoring assessments allow school professionals 
to evaluate the efficacy of interventions and enable data-based decision making concerning 
supports for specific students. 
The Need for Formative Measures of Student Social Behavior 

Briesch and Volpe (2007) provide a conceptual framework for evaluating progress 
monitoring assessments that balances considerations of psychometric adequacy and feasibility. 
Ideally, such formative measures should accurately measure the intended behavior or construct, 
be sensitive to changes in student behavior, and be feasible for school professionals to implement 
in terms of training, time, and cost. Unfortunately, the current knowledge base regarding 


behavioral progress monitoring measures is limited relative to those available for academic 
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progress monitoring (Chafouleas, Volpe, Gresham, & Cook, 2010). Whereas the National Center 
on Intensive Intervention (NCII; n.d.) Academic Progress Monitoring Tools Chart summarizes 
the psychometric evidence for 25 tools with multiple subtests assessing various academic skills 
and grade levels, only six tools are listed in the Behavioral Progress Monitoring Tools Chart. The 
chart includes two behavior rating scales (Behavior Intervention Monitoring Assessment System 
[BIMAS; McDougal, Bardos, & Meier, 2011] and the Behavioral Assessment System for 
Children-2 Progress Monitor [BASC-2 Progress Monitor; Reynolds & Kamphaus, 2009]), 
observational assessment of academic engagement (i.e., momentary time sampling for academic 
engagement) and two Direct Behavior Rating (DBR) measures (i.e., academically engaged, 
disruptive behavior). Unfortunately, each of these currently available progress monitoring tools 
demonstrates limitations in regard to either feasibility, psychometric adequacy, or both. 
Although rating scales are one of the most commonly used tools by school psychologists 
(Shapiro & Heick, 2004), most have been constructed for use in diagnostic assessments and as 
such were designed to assess symptoms of common child psychiatric disorders as opposed to 
socially valid and malleable indicators of student academic and social functioning (Pelham, 
Fabiano, & Massetti, 2005). Rating scales developed primarily for such summative assessments 
may require extensive time for completion, are restricted in regard to the range of behavioral 
constructs assessed, and may be inappropriate for measuring short-term changes in student 
behavior in response to classroom intervention (Volpe & Gadow, 2010). Finally, none of the 
rating scales identified by NCII provide “convincing” criterion-related validity and treatment 
sensitivity evidence. Without evidence supporting desirable psychometric characteristics or 
feasible implementation, extant rating scales would seem to be of limited utility for progress 


monitoring purposes in MTSS models. 
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Another method identified by NCII is observational assessment of academic engagement 
via momentary time sampling (MTS). Evidence indicates that MTS generates reliable 
measurement of student engagement behavior (Briesch, Chafouleas, & Riley-Tillman, 2010; 
Hintze & Matthews, 2004; Wood, Hojnoski, Laracy, & Olson, 2015; Volpe, DiPerna, Hintze, & 
Shapiro, 2005). However, to achieve acceptable levels of dependability for progress monitoring 
(e.g., .70) requires trained observers to conduct two 30-minute, three 15-minute, or four to five 
10-minute observations on each targeted student (Briesch, Volpe, & Ferguson, 2014; Ferguson, 
Briesch, Volpe, & Daniels, 2012). Furthermore, MTS data are highly context-specific, which can 
limit the generalizability of resultant data (Chafouleas et al., 2010). Although MTS for academic 
engagement demonstrates favorable psychometric characteristics, the time required for training 
accurate observers and the time needed to conduct observations limit its utility as a progress 
monitoring measure in MTSS models (Volpe & Gadow, 2010; Volpe, Gadow, Blom-Hoffman, 
& Feinberg, 2009). 

A third class of progress monitoring assessment identified by NCII, DBR, has been 
described as a hybrid tool combining elements of systematic direct observation with those of 
behavior rating scales (Chafouleas, 2011). DBR typically are completed by classroom teachers, 
and unlike systematic direct observation do not require trained external observers. DBR may 
minimize inference and retrospective judgments by directly rating observable, operationally 
defined behaviors in close temporal proximity and in naturalistic settings (Chafouleas, Riley- 
Tillman, & Christ, 2009; Chafouleas, Riley-Tillman, & Sugai, 2007; Christ, Riley-Tillman, & 
Chafouleas, 2009). Two primary methods of DBR have been investigated. Single-Item Scales 
(DBR-SIS) typically measure a global construct (e.g., academic engagement) using a single item. 


In contrast,Multi-Item Scales (DBR-MIS) are comprised of several items (typically between 


DBR-MIS ENABLER SCALES 6 


three and five items) that assess specific behaviors, the scores of which can be summed to obtain 
a composite score measuring the construct of interest (e.g., Volpe & Briesch, 2012). The 
majority of DBR research has focused on DBR-SJ) ratings of Academic Engagement, 
Respectful, and Disruptive Behavior (Briesch, Chafouleas, & Riley-Tillman, 2016; Chafouleas, 
2011). Studies have demonstrated the reliability and dependability (Chafouleas et al., 2010; 
Chafouleas, Christ, Riley-Tillman, Briesch, & Chanese, 2007), validity (Riley-Tillman, 
Chafouleas, Sassu, Chanese, & Glazer, 2008), and treatment sensitivity (Chafouleas, Sanetti, 
Kilgus, & Maggin, 2012; Fabiano, Pyle, Kelty, & Parham, 2017) of DBR-SIS assessing 
Academic Engagement and Disruptive Behavior. 

The use of DBR-SIS of Academic Engagement and Disruptive for progress monitoring 
assessment offers many advantages (e.g., validity, reliabilitiy, feasibility); however the 
restriction of SEB progress monitoring assessment to only two constructs is very narrow. 
Although the relationship between academically engaged and disruptive student behavior and 
academic achievement is well documented (e.g., Atkins, Hoagwood, Kutash, & Seidman, 2010; 
Reyes, Brackett, Rivers, White, & Salovey, 2012), a broader set of constructs have been 
identified as having a positive impact on academic achievement (e.g., DiPerna & Elliott, 1999) 
and represent appropriate targets for classroom interventions (e.g., DiPerna & Elliott, 2002; 
Volpe et al., 2006). Thus, an expanded set of progress monitoring tools that assess a wider range 
of such positive behaviors is needed. 

Academic Enablers 

In the current study, we report the intial development of DBR-MIS to assess an expanded 

set of academic enabler constructs. As defined by DiPerna and Elliott (1999), academic 


competence is “a multi-dimensional construct composed of the skills, attitudes, and behaviors of 
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a learner that contribute to teachers’ judgments of academic performance” (p. 208). Academic 
skills alone are therefore insufficient for academic success; rather, students must also 
demonstrate adequate levels of engagement and possess adequate interpersonal skills, study 
skills, and academic motivation (DiPerna et al., 2002). Academic engagement reflects the degree 
to which a student sustains attention and participates in the curriculum. Although the construct 
encompasses more than simply attending behavior, the relationship between inattention and 
academic outcomes is clear (e.g., Greenwood, 1996; Massetti et al., 2008), even when 
controlling for cognitive and other behavioral variables (Rabiner & Coie, 2000). 

The construct of interpersonal skills encompasses the ability to effectively communicate 
and cooperate with both peers and adults. Interpersonal skills have been shown to have 
significant direct and indirect effects on student achievement (Henricsson & Rydell, 2006; 
Malecki & Elliott, 2002; Wentzel, 1998). According to the models proposed by DiPerna and 
Elliott (2002), student interpersonal skills interact with prior achievement and predict motivation, 
which in turn has significant effects on two direct predictors of current achievement: study skills 
and engagement. 

Finally, study skills refer to behaviors that facilitate the process of learning new 
information, including taking notes, checking work, following directions, and organizational 
behaviors such as keeping one’s materials/desk organized and remembering materials and 
assignments (DiPerna & Elliott, 2000). Investigations have shown that study skills are a 
significant, direct predictor of academic achievement across grade levels and are common targets 
for academic intervention (Gettinger & Seibert, 2002; Harvey & Chickie-Wolfe, 2007; Jenkins & 


Demaray, 2015; Volpe, DuPaul, DiPerna, & Jitendra, 2006). 
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Several longitudinal studies have demonstrated the importance of academic enablers in 
predicting academic achievement for children and adolescents (e.g., DiPerna & Elliott, 1999; 
DiPerna, Volpe, & Elliott, 2002; DiPerna, Volpe, & Elliott, 2005; Reynolds & Walberg, 1992; 
Volpe et al., 2006). Expanding formative behavioral assessment to include academic enablers is 
also critical as such behaviors are frequent targets of school-based interventions for students with 
learning and behavior problems (e.g., Barkley, 2016; Fabiano et al., 2010). 

Purpose of Study 

Heretofore, DBR assessment of academic enablers has been limited to the construct of 
academic enagement (e.g., Chafouleas et al., 2010; Volpe & Briesch, 2012, 2015). Given than 
academic enablers are both critical to academic success and appropriate targets of classroom 
intervention, we sought to expand the breadth of formative assessment of academic enablers. The 
purpose of current study was to develop and evaluate teacher-completed DBR-MIS scales to 
assess the academic enabler constructs of Academic Engagement, Interpersonal Skills, and Study 
Skills in early elementary school students. First, the content validity of an initial pool of 
candidate items for each scale was evaluated by a panel of consumers (e.g., school professionals 
and parents) and a panel of researchers with expertise in school-based behavioral assessment. 
The construct validity of the three proposed scales was then subsequently evaluated through 
exploratory factor analysis (EFA). Finally, we examined the internal consistency of each novel 
scale. 

Method 
Item Development 
Item content for the Academic Engagement, Interpersonal Skills, and Study Skills scales 


was developed through a four-stage process. In the first stage, items were drafted based on three 


DBR-MIS ENABLER SCALES 9 


sources of information: a) a national teacher survey of common referral concerns (Briesch et al., 
2013), b) a review of extant rating scale and observation measures assessing the constructs of 
interest, and c) a review of frequently used targets for Daily Report Card interventions (Owens et 
al., 2012). 

In the second stage, a Consumer Advisory Panel (CAP) consisting of four kindergarten 
through grade 3 (K-3) general education classroom teachers, one special education teacher, one 
elementary school principal, two school psychologists, and four parents of children in K-3 
provided feedback on the aforementioned items. Members of the CAP were sent a link to provide 
feedback on items using Qualtrics software (Qualtrics, Provo, UT). Members of the CAP rated 
each candidate item using a 5-point Likert-type scale (0 = Strongly Disagree; 4 = Strongly 
Agree) on three criteria: (a) observability (they would be able to see it in a typical classroom 
setting), (b) malleability (it would be a suitable target for classroom intervention), and (c) social 
validity (if it improved, it would be helpful to the student in question or the classroom 
environment at large). Members of the CAP also provided feedback regarding clarity in the 
wording of items and were asked to list any additional behavioral targets that should be 
considered in the assessment of the constucts of interest. Both qualitative and quantitative 
feedback from the CAP was used to inform modifications to the composition of existing items 
before moving on to the next stage of development. 

In the third stage, a Scientific Advisory Panel (SAP) comprised of five scholars with 
expertise in school-based behavioral assessment and scale development, the constructs of 
interest, and statistical methods reviewed and rated the items on a 5-point Likert scale (0 = 


Strongly Disagree; 4 = Strongly Agree) according to the following criteria: (a) construct 
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relatedness, (b) observability, (c) sensitivity to treatment, and (d) social validity. The SAP also 
provided feedback on wording of items, content coverage and any other suggestions for revision. 

In the fourth stage, resultant items were administered to a large sample of elementary 
school teachers. Teacher ratings for each item were submitted to separate EFA for each scale in 
order to empirically identify items most representative of each of the three academic enabler 
constructs. Initial item retention decisions were based on the results of the EFA and previous 
ratings from the CAP and SAP. Subsequently, the SAP conducted a final review to determine 
which items should be retained for each scale. 

Exploratory Factor Analysis 

Participants. Participants were general- and special-education teachers from 35 school 
public school districts across 13 states. A total of 307 K-3 teachers each completed ratings for 
one randomly selected student in their class. Teachers were primarily female (95.8%) with a 
wide range of teaching experience (see Table 1). The student sample was comprised of 187 
(60.9%) males and 120 (39.1%) females, and students were nearly evenly distributed across 
grade levels (see Table 2). The composition of students by race and ethnicity was as follows: 
67.1% White, 13% Black, 15.0% Hispanic, 3.3% Asian, 1.0% American/Alaska Native, and 
5.2% Unknown. Approximately 35% of students were receiving special education services at the 
time of data collection. 

Measures. Teachers were instructed to rate the behavior of target students over the past 5 
school days using a 7-point frequency scale ranging from Never = 0 to Almost Always = 6. Items 
for each academic enabler scale were developed through the 4-stage process described above. 

Data collection procedures. Each participating K-3 teacher rated one randomly 


identified student from his/her class. Teachers were assigned a randomly generated number 
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between | and 20 and selected the corresponding student from his/her alphabetical class roster. 
The random selection of the student was designed to prevent any score variance restriction. 
Researchers created and adhered to a standardized protocol to ensure similar procedures during 
each administration. Participants were sent a link to an online portal to complete the ratings, 
which the authors created using Qualtrics software (Qualtrics, Provo, UT). This format prevented 
missing data by requiring all items to be rated for submission. Participants completed ratings for 
only one student to ensure the independence of ratings. 

Data analysis. Exploratory factor analysis is inherently theory-driven and can be used as 
a confirmatory approach (McDonald, 1999). As such, EFA was performed separately for each of 
the three scales to identify items that would serve as strongest indicators of the relevant latent 
construct identified a priori. As items were developed to measure their corresponding construct, 
all items within each DBR-MIS scale were expected to load substantially on a single factor. 
Principal axis factoring (PAF) was used as the factor extraction method because it does not 
assume multivariate normality (Fabrigar, Wegener, MacCallum, & Strahan, 1999). Parallel 
analysis (PA; Horn, 1965) was conducted to inform factor retention decisions within each scale. 
PA permutated multiple correlation matrices of raw data and computed eigenvalues. A factor 
with an eigenvalue higher than the 95" percentile of all eigenvalues was retained (Velicer, Eaton, 
& Fava, 2000). In cases where more than one factor was identified within a scale, direct oblimin, 
an oblique rotation method, was adopted due to the expected correlations between factors 
(Osborne & Costello, 2009). 

Results 


Item Development 
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Following initial item generation from relevant research (Briesch et al., 2013; Owens et 
al., 2012) and extant measures (e.g, Academic Competence Evalutaion Scales [ACES], Behavior 
Rating Inventory of Executive Function [BRIEF]), observation codes (e.g., Achenbach System 
of Empirically Based Assessment: Direct Observation Form), the CAP provided feedback on an 
initial pool of 41 potential DBR-MIS items (13 for Academic Enagement, 20 for Interpersonal 
Skills and 8 for Study Skills). Ranges of mean CAP ratings varied across constructs and across 
the three dimensions rated by the CAP. For Academic Engagement, ratings of observability 
ranged from 2.50 to 3.91, ratings of malleability ranged from 1.50 to 3.08, and ratings of social 
validity ranged from 1.83 to 3.25. For Interpersonal Skills, ratings ranged from 2.25 to 3.58 
(observability), from 2.17 to 3.42 (malleability), and from 2.42 to 3.42 (social validity), 
respectively. For Study Skills CAP ratings ranged from 2.25 to 3.55, from 2.00 to 3.33, and from 
2.08 to 3.58 for observability, malleability and social validity, respectively. In response to 
feedback from the CAP we added an additional interpersonal skills item (appropriately 
maintains interactions with peers). 

Next, the SAP provided feedback on all items. Ranges of mean SAP ratings for each item 
across raters varied across constructs and across the four dimensions. For Academic 
Engagement, ratings of criterion relatedness and observability both ranged from 2.8 to 4.0, 
ratings of malleability ranged from 2.0 to 4.0 and ratings of social validity ranged from 2.8 to 
4.0. For Interpersonal Skills, ratings of criterion-relatedness ranged from 2.6 to 4.0, ratings of 
observability ranged from 2.0 to 3.8, ratings of malleability ranged from 2.4 to 3.8, and ratings of 
social validity ranged from 2.8 to 3.8. Ratings for Study Skills items were consistently higher 
(criterion relatedness range = 3.4-4.0; observability range = 3.2-4.0; malleability range 3.6-4.0, 


and social validity range = 3.6-4.0). 
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A total of five items were deleted because mean ratings from the CAP, the SAP, or both 
were below 3. Of these deleted items, three were designed to measure interpersonal skills 
(appreciates others, shows empathy, and approporiately maintains interactions with peers) and 
two were designed to measure academic engagement (sticks with tasks until complete, and shows 
enthusiasm for lesson). An additional item (runs out of time before assignments are complete) 
was deleted because it was considered to be redundant with another item (finishes work on time). 
A total of 35 items were submitted to EFA. 

Exploratory Factor Analysis 

As indicated above, data collection procedures required teachers to rate every item of the 
three novel scales in order to submit their responses. Descriptive statistics indicated that the 
distribution of some items on each scale were skewed or kurtotic. All items were within normal 
limits (i.e., ratio < +3) for skewness on the Academic Engagement (range = -2.47 - 2.15) and 
Interpersonal Skills (range = -2.20 - 0.79) scales but not on the Study Skills scale (range = -5.19 - 
3.21) whereas some items were kurtotic on all scales (Academic Engagement range = -4.28 - - 
2.31; Interpersonal Skills range = -4.29 - -2.42; Study Skills range = -4.23 - -2.33). Given the 
variable directions and severity in skewness and kurtosis for items within each scale, data 
transformations were not performed as they would have exacerbated skewness and kurtosis for 
some items. 

Inter-item Pearson correlation coefficients within each scale were sufficiently high to 
conduct EFA, but not so high as to indicate the potential for multicollinearity (i.e., rs were 
generally < .90). Bartlett's Test of Sphericity was significant (p < .001) for all scales, which 
indicated the correlation matrix was not an identity matrix for each scale, and the following 


Kaiser-Meyer-Olkin Measure of Sampling Adequacy values were obtained: Academic 
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Engagement = .95, Interpersonal Skills = .97, and Study Skills = .93. Decisions on item 
reduction were made on the basis of multiple considerations beyond factor loadings including 
ratings from the CAP and SAP. 
Academic Engagement 

The initial Academic Engagment scale consisted of 10 items. The range of inter-item 
correlations was .58-.84 (see Table 3) with base rates from 20.5 to 52.8% (see Table 4). Parallel 
Analysis indicated a one-factor solution, which explained 73.41% of the observed variance with 
the overall mean factor loading of the solution was .86 (range = .75-.92). It should be noted that 
all factor loadings fell within the excellent range (> .71; Comrey & Lee, 1992). A total of five 
items were selected for retention and the remaining items were removed (see Table 4). 
Coefficient alpha for the 5-item scale Academic Engagement scale was .94. 
Interpersonal Skills 

The initial Interpersonal Skills scale consisted of 17 items. The range of inter-item 
correlations was .545 to .885 (see Table 5) with base rates from 18.2 to 42.7% (see Table 6). 
Parallel Analysis indicated a two-factor solution. However, the second factor contributed little 
additional variance (4.78%) and was not conceptually clear. Thus, a single factor solution was 
forced. This factor explained 71.46% of the observed variance. The overall mean factor loading 
for items in the one factor model was .84 (range = .75 - .91). A total of five items were retained 
as shown in Table 6. Coefficient alpha for the 5-item Interpersonal Skills scale was .93. 
Study Skills 

The initial Study Skills scale consisted of 8 items. The range of inter-item correlations 
was .465-.834 (see Table 7) with base rates from 21.5 to 58.3% (see Table 8). Parallel Analysis 


indicated a one-factor solution, which explained 67.10% of the observed variance. The overall 
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mean factor loading of the solution was .811 (range = .614-.914; see Table 8). A total of 5 items 
were retained for this scale. The factor loading of one item (completes homework) fell below the 
excellent range of magnitude (i.e. .61), but was still considered acceptable (Stevens, 1992; 
Tabachnick & Fidell, 2007). Coefficient alpha for the 5-item Study Skills scale was .898. 

Correlations between composite scores generated from summing retained items on each 
scale were high. Correlations between Academic Engagement and Interpersonal Skills and Study 
Skills were .71 and .90 respectively and the correlation between Interpersonal Skills and Study 
Skills was .71. 

Discussion 

In this article, we report the intial development of DBR-MIS designed to measure the 
three academic enabler constructs of Academic Engagement, Interpersonal Skills, and Study 
Skills. The current study builds upon extant work in the formative assessment of student 
behavior in several important ways. First, it broadens the assessment of academic enabler 
constructs beyond Academic Engagement (e.g., Chafouleas et al., 2010; Volpe & Briesch, 2012, 
2015) to include the formative assessment of both Interpersonal Skills and Study Skills. Second, 
the study employed multiple methods to generate an appropriate pool of items for each construct 
and is among the first studies to apply EFA methodology to DBR assessment. 

Based on the methodology employed by Volpe and Briesch (2012), we collected data 
from stakeholders (teachers, parents, school administrators and research scientists) regarding 
criterion-relatedness, observability, malleability and social validity to ensure items generated to 
measure each academic enabler construct were appropriate for formative assessment in MTSS 
models. We believe that such screening of items is critically important in the development of 


formative measures of student behavior and should serve as a model for content validity studies 
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in DBR assessment. That is, it is not enough to know that specific behaviors are related to a 
particular construct. In the context of formative assessment in MTSS models, wherein ratings are 
completed often (e.g., daily or weekly) to evaluate a student’s response to intervention, it is also 
important to know that a) the behavior can be readily observed during the relevant observation 
interval in the setting of interest (e.g., during classroom instruction on a typical school day; 
Volpe, McConaughy, & Hintze, 2009), b) it is a behavior that would be expected to change in 
response to evidence-based intervention (see Gresham et al., 2010), and c) if it did change, it 
would be of benefit to the student (Wolf, 1978). 

We conducted a series of EFAs to identify items that would be the strongest indicators of 
each academic enabler construct. Results indicated one-factor solutions for both the Academic 
Engagement and Study Skills constructs. The EFA for Interpersonal Skills items initially 
generated a two-factor solution, but because the second factor contributed little incremental 
variance and the grouping of items across the two factors was not conceptually clear, we 
examined a single-factor model. Across EFAs, items with relatively low base rates and/or low 
factor loadings in comparison to other items in the same scale were removed, which resulted in 
5-item scales for each of these constructs. The single factor models for each of the three 
academic enabler scales accounted for substantial amounts of variance (between 67.1 and 73.4%) 
in teacher ratings and each DBR-MIS demonstrated adequate internal consistency (alphas 
between .898 and .939). 

Ours is the first study to examine the factor structure of items designed specifically for 
progress monitoring of academic enablers and for rating observations conducted over relatively 
short observation intervals (days or a single week). However, in a recent study, Anthony and 


DiPerna (2018) conducted an confirmatory factor analysis (CFA) of a short form of the ACES 
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that included scales similar to those examined in the current study (i.e., Academic Engagement, 
Interpersonal Skills, and Study Skills). Like many traditional rating scales, directions for the 
ACES do not specifiy a specific time frame from which informants should base their ratings. 
Maximally efficient items were drawn from the original teacher-completed ACES (DiPerna & 
Elliott, 2000) based on item response theory (Anthony & DiPerna, 2017). Results of the CFA 
were consistent with results of prior studies of the full-length version of the ACES supporting a 
single tiered structure with correlated factors. This was consistent with the models examined in 
the current study. However, correlations between the three enabler scales developed in the 
current study were high (between .71 and .90). Correlations between scales on our newly 
developed DBR are notably higher than those found for corresponding scales on the ACES in 
prior studies (e.g., DiPerna et al., 2002, 2005). Although the content of the DBR evaluated in the 
current study was screened to correspond to the constructs assessed by the ACES, these larger 
associations between measures assessing related constructs may be the results of our focus on 
only the most observable and malleable behaviors and ratings conducted based on relatively 
short observation intervals. Most notably, the correlation between the Academic Engagement 
and Study Skills scales was much higher than expected. Although one would expect students 
with high levels of engagement to also demonstrate the effective study habits assessed by the 
Study Skills scale, to the extent that particular classroom interventions might result in differential 
intervention response across these two groups of behaviors, it may be useful to maintain these 
two different scales for the time being. 
Limitations 

Although the findings of the current study yielded favorable psychometric characterisitcs 


for the novel academic enabler scales concerning construct validity and internal consistency, 
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these findings should be considered in light of several limitations. First, although we successfully 
recruited teachers from 13 states and students rated by these teachers were selected at random 
from class lists, White students and males were over-represented (U.S. Department of Education, 
2018). As such, the results of the current study may not generalize to samples with a higher ratio 
of students from ethnically diverse backgrounds or samples with a larger proportion of female 
students. Although the demographics of our participating teachers are fairly consistent with those 
of teachers in the United States (U.S. Department of Education, 2018), it is not clear to what 
extent these findings would generalize to samples with a higher proportion of male teachers. 
Although the size of the sample would be considered adequate for our EFA, due to the 
composition of the sample in regard to demographic characteristics, it is not well-suited to tests 
of measurement invariance (e.g., Meade & Bauer, 2007). As such, the aforementioned 
limitations could not be addressed in the present study. 
Future Directions 

The current study represents an initial step in a larger scale process of scale development 
and evaluation. Additional studies currently are being conducted to address the dependability, 
treatment sensitivity, concurrent and discriminant validity, and acceptability of these novel 
measures. In the next phase, the dependability of these DBR-MIS will be evaluated through a 
series of generalizability and dependability studies to identify optimal methods for obtaining 
dependable progress monitoring data. Treatment sensitivity will be examined by evaluating the 
extent to which the scales detect changes in student behavior in response to evidence-based 
classroom intervention (i.e., Daily Behavior Report Card; Volpe & Fabiano, 2013). Future 
studies also will examine the concurrent validity of these novel scales, using established 


traditional rating scales and systematic direct observation as criterion measures. In sum, these 


DBR-MIS ENABLER SCALES 19 


studies are designed to provide the evidence needed to utilize this broadened set of tools for the 


formative assessment of academic enablers in school-based problem-solving models. 
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Table 1. 


Teacher Demographics 


Frequency Percentage of Sample 

Gender 

Male 13 4.2% 

Female 294 95.8% 
Years of Experience 

0-5 73 23.8% 

6-10 61 19.9% 

11-20 107 34.9% 

21-30 48 15.6% 


> 30 18 5.9% 
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Table 2. 


Student Demographics 


30 


Frequency Percentage of Sample 
Gender 
Male 187 60.9% 
Female 120 39.1% 
Race 
White 206 67.1% 
Black 40 13.0% 
Latino 46 15.0% 
Asian 10 3.3% 
American/Alaska Native 3 1.0% 
Hawaiian/Pacific Islander 1 0.3% 
Unknown 16 5.2% 
Age (Years) 
5 26 8.5% 
6 67 21.8% 
i] 71 23.1% 
8 82 26.7% 
9 55 17.9% 
10 > 1.6% 
Grade 
Kindergarten 82 26.7% 
First 64 20.8% 
Second 79 25.7% 
Third 82 26.7% 
Special Education Status 
Receiving Special Education 109 35.5% 
Not Receiving Special Education 196 63.8% 
Unsure 2 0.7% 
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Table 3. 


Motivation/Engagement Inter-Item Correlations 
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Item* 1 2 3 4 5 6 7 8 9 10 
1. Interest in task — 

2. Independent ae. — 

3. Avoids being distracted .69 76 — 

4. Starts promptly 16 cl} 79 — 

5. Works hard 81 fe: i ie — 

6. Raises hand .65 .68 .68 .70 .69 — 

7. Adequate progress 74 75 a2 82 WD .60 — 

8. On time .74 .74 .68 82 oe 58 .84 — 

9. Actively participates ee .67 61 Ag he: ork .63 .69 3 — 

10. On task J We) .86 85 81 .65 719 wii .68 — 


Note. * = abbreviated description of item. 
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Table 4. 


Engagement Item Means, Standard Deviations, Base Rates, and Factor Loadings 


Item* Mean SD ae ae 
1. Interest in task* Sh 1.45 20.5% 862 
2. Independent 3.54 1.79 31.6% 859 
3. Avoids being distracted* 2.61 1.76 52.8% 848 
4. Starts promptly 3.34 1.76 37.8% 919 
5. Works hard* 3.91 1.54 23.1% 877 
6. Raises hand* 3.73 1.67 24.4% 749 
7. Adequate progress* 3.65 1.64 30.3% 871 
8. On time 3.54 1.80 35.2% .860 
9. Actively participates 3.84 1.59 25.4% .800 
10. On task 330 1.62 35.8% 912 


Note. “= abbreviated description of item. * = item removed 
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Table 5. 

Interpersonal Skills Inter-Item Correlations 

Item* 1 2 3 4 5 6 7 8 9 10 Il 06120613145 16 
1. Shows interest — 

* ae se 

3. Compromises 69 67 —- 

4. Expresses feelings .63 65 69 — 

5. Helps 77 78 68 67 - 

es 2) 8 FS BO = 

7. Respectful 67 69 3: 267 «71 369 = 

8. Cooperates to at2  af® Jl 474. .7F . 84 — 

9. Listens 78 10 .70 64 72 .71 70) 74 

10. Contributes ato 260" 625: -63¢- 39S. a7. 2.62. 472° SID< = 

11. Shares 64 63 67 64 73 73 65 71) 6606 (68 CK 

12. Polite 67 68 73 67 68 .70 89 81 68 62 66 —- 

13. Plays well 68 68 72 .71 74 80 80 85 .70 68 69 79 - 

14. Kind 68 72 68 68 73 .71 85 6.810669) «62006 (67) (8606 BL CK 

15. Asks for help 63 62 54 69 71 68 62 64 65 .70 59 60 62 60 —- 

16. Works well Pl !- 309 . aS 269 7A SKE 82° ORT 2) AI. 369% B08 86. 279. GR ~ 
17. Takes turns 4 1. TS 68 ATS ID TT BB. TB. TL. TS TT B82 TS 68 BO 
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Note. “= abbreviated description of item 
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Table 6. 


Interpersonal Skills Item Means, Standard Deviations, Base Rates, and Factor Loadings 


Item* Mean SD Base Rate pee 
1. Shows interest* 3.64 1.505 27.4% .828 
2. Positive comments* 3.44 1.588 32.6% .818 
3. Compromises 3.00 1.730 42.7% 820 
4. Expresses feelings 3.39 1.682 33.6% 192 
5. Helps* 3.63 1.582 26.4% .864 
6. Initiates interactions 3.66 1.601 27.0% 871 
7. Respectful 4.06 1.520 19.2% 871 
8. Cooperates 3.76 1.568 26.4% 912 
9. Listens* 3.68 1.476 26.7% .832 
10. Contributes* 3.64 Lo 26.4% 804 
11. Shares* 3.91 1.534 21.8% .793 
12. Polite* 4.10 1.514 18.9% .860 
13. Plays well* 3.94 1.598 22.1% 888 
14. Kind* 4.14 1.523 18.2% .863 
15. Asks for help* 3.63 1.632 26.4% 746 
16. Works well* 3.80 1.604 25.1% 895 
17. Takes turns* 3.87 1.523 21.5% 893 


Note. “= abbreviated description of item. * = item removed 
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Table 7. 

Study Skills Inter-Item Correlations 

Item* 1 2 3 4 5 6 7 
1. Checks work — 

2. Accurate .69 = 

3. Well organized AD 71 — 

4. Prepared 76 .69 82 - 

5. Writes down 61 AT 58 61 — 

6. Keeps track 74 61 85 83 59 — 

7. Follows instructions 78 ae AAS) 81 56 78 — 
8. Completes 51 49 re 58 50 ol AT 


Note. * = abbreviated description of item. 
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Tab 


Study Skill Item Means, Standard Deviations, Base Rates, and Factor Loadings 


le 8. 


Item* Mean SD Base Rate ee 
1. Checks work* 2.48 1.77 58.3% .866 
2. Accurate 3.66 1.56 26.1% .764 
3. Well organized 3.08 1.87 42.3% .907 
4. Prepared* SiS 1.72 29.0% 914 
5. Writes down* 239 211 53.7% .666 
6. Keeps track 3.33 1.87 35.8% .883 
7. Follows instructions 3.67 1.56 26.1% .880 
8. Completes 4.13 1.96 21.5% .614 
Note. “= abbreviated description of item. * = item removed 


